1. (Previously Presented) A system for synchronizing multi-modal interactions, 
comprising: 

a program storage device that stores a multi-modal application, the multi-modal 
application comprising at least a first mode process that enables user interaction with the 
application in a first modality and a second mode process that enables user interaction with the 
application in a second modality; 

a program execution system that executes the multi-modal application and synchronizes 
the first and second mode processes while a user interacts with the multi-modal application, 
wherein the program execution system comprises: 

a multi-modal shell that manages information exchanges between the first and second 
mode processes of the multi-modal application to enable a synchronized multi-modal interaction 
with the multi-modal application wherein user interaction in one modality resuhs in execution of 
corresponding commands in both the first and second mode processes; and 

an API (application program interface) that allows the first and second mode processes to 
register their respective active commands and corresponding actions with the muhi-modal shell. 

2. (Previously Presented) The system of claim 1, wherein the program execution system 
comprises a registry having a registration table, managed by the multi-modal shell, wherein the 
registration table comprises a list of each of the registered commands and corresponding 
synchronized actions that results in both the first and second mode processes upon execution of a 
registered command by one of the first and second mode processes. 

3. (Previously Presented) The system of claim 1, wherein the multi-modal application 
comprises at least a first mono-mode application for the first mode process and a second mono- 
mode application for the second mode process, wherein the multi-modal shell manages and 
synchronizes information exchanges between the first and second mono-mode applications. 

4. (Original) The system of claim 1, ftirther comprising at least a first device having a 
first user interface modality and a second device having a second user interface modality, 
wherein the multi-modal shell manages and synchronizes information exchanges between the 
first and second devices. 
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5. (Original) The system of claim 4, wherein the first device, second device and multi- 
modal shell are distributed over a network, and wherein the API is implemented using distributed 
APIs or protocols. 

6. (Original) The system of claim 2, wherein the API comprises a mechanism for 
converting a mono-mode application to a multi- modal application. 

7. (Original) The system of claim 6, wherein the mono-mode application is a GUI 
application, and wherein the mechanism provides speech enablement of the GUI application by 
registering the active commands of the GUI application and building a grammar for the 
registered commands to support the commands in a speech modality. 

8. (Original) The system of claim 2, wherein the API comprises a mechanism for building 
a multi-modal application. 

9. (Original) The system of claim 8, wherein the mechanism is used for directly 
programming the registry by building a registration table having user-defined commands and 
corresponding actions for each of the modalities of the multi-modal application. 

10. (Original) The system of claim 1, further comprising an operating system, wherein 
the multi-modal shell executes on top of the operating system. 

1 1 . (Original) The system of claim 1, wherein the system is distributed over a network. 

12. (Previously Presented) The system of claim 1, wherein the multi-modal application is 
a multi-modal browser comprising a first browser application and a second browser application. 

13. (Previously Presented) The system of claim 12, wherein the first browser is a GUI 
browser and the second browser is a speech browser. 
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14. (Previously Presented) The system of claim 12, wherein the multi-modal shell 
processes the multi-modal application to send modality-specific presentation information to the 
respective browsers. 

15. (Previously Presented) The system of claim 14, wherein the multi -modal application 
is authored using a modality-independent description and wherein the multi-modal shell 
generates the modality-specific presentation information from the modality-independent 
description. 

16. (Previously Presented) The system of claim 14, wherein the multi-modal application 
comprises a combination of declarative markup languages to describe the first and second mode 
processes. 

17. (Previously Presented) The system of claim 16, wherein the multi-modal application 
combines the declarative markup languages and synchronization elements in single pages to 
provide tight synchronization between the first and second mode processes. 

18. (Previously Presented) The system of claim 16, wherein the multi-modal application 
comprises a separate files of the declarative markup languages for the first and second mode 
processes and wherein the separate files are loosely synchronized at predefined points. 

19. (Previously Presented) A method for synchronizing multi -modal interactions, 
comprising the steps of: 

activating a multi-modal application comprising at least a first mode process that enables 
user interaction with the application in a first modality and a second mode process that enables 
user interaction with the application in a second modality 

receiving a command or event in the first modality; 

triggering an action by the first mode process based on the received command or event 
and triggering a corresponding action by the second mode process; and 

updating application states or device states associated with the first and second mode 
processes. 
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20. (Previously Presented) The method of claim 19, further comprising the steps of: 
registering active commands associated with the first mode process and active conmiands 

associated with the second mode process; 

associating, with each registered command of the first mode process, an action on the 
first mode process and a corresponding action on the second mode process; and 

associating, with each registered command of the second mode process, an action on the 
second mode process and a corresponding action on the first mode process. 

21. (Original) The method of claim 20, further comprising the step of building a 
command-to-action registration table based on the registered commands and actions. 

22. (Previously Presented) The method of claim 21, wherein the registration table is 
built by a multi-modal shell via API calls from the applications or devices associated with the 
first and second mode processes. 

23. (Previously Presented) The method of claim 20, wherein the step of triggering 
comprises the steps of: 

looking up the received command in the registration table; and 

executing the actions associated with the received command by the first and second mode 
processes. 

24. (Previously Presented) The method of claim 20, further comprising the steps of: 
registering a callback handle for each of the registered commands to notify the first and 

second mode processes of completion of the actions corresponding to the registered commands. 

25. (Previously Presented) The method of claim 24, wherein the step of updating the 
application states or the device states comprises the steps of executing the callback handle 
associated with the received command to trigger a callback action on the first mode process and 
a callback action on the second mode process. 
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26. (Previously Presented) The method of claim 19, wherein the step of triggering 
comprises the steps of: 

executing first thread associated with the received command; and 
triggering a corresponding second thread to initiate the corresponding action by the 
second mode process. 

27. (Original) The method of claim 26, wherein the threads are applets. 

28. (Original) The method of claim 26, wherein the threads communicate via socket 
connections. 

29. (Previously Presented) A program storage device readable by a machine, tangibly 
embodying a program of instructions executable by the machine to perform method steps for 
synchronizing multi- modal interactions, the method comprising the steps of: 

activating a multi-modal application comprising at least a first mode process that enables 
user interaction with the application in a first modality and a second mode process that enables 
user interaction with the application in a second modality 

receiving a command or event in the first modality; 

triggering an action by the first mode process based on the received command or event 
and triggering a corresponding action by the second mode process; and 

updating application states or device states associated with the first and second mode 
processes. 

30. (Previously Presented) The program storage device of claim 29, further comprising 
instructions for performing the steps of: 

registering active commands associated with the first mode process and active commands 
associated with the second mode process; 

associating, with each registered command of the first mode process, an action on the 
first mode process and a corresponding action on the second mode process; and 

associating, with each registered command of the second mode process, an action on the 
second mode process and a corresponding action on the first mode process. 
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31 . (Original) The program storage device of claim 30, further comprising instructions 
for performing the step of building a command-to-action registration table based on the 
registered commands and actions. 

32. (Previously Presented) The program storage device of claim 30, vv^herein the 
instructions for performing the step of triggering comprise instructions for performing the steps 
of: 

looking up the received command in the registration table; and 

executing the actions associated with the received command by the first and second mode 
processes. 

33. (Previously Presented) The program storage device of claim 30, further comprising 
instructions for performing the steps of: 

registering a callback handle for each of the registered commands to notify the first and 
second mode processes of completion of the actions corresponding to the registered commands. 

34. (Previously Presented) The program storage device of claim 33, v^herein the 
instructions for performing the step of updating the application states or the device states 
comprise instructions for performing the step of executing the callback handle associated with 
the received command to trigger a callback action on the first mode process and a callback action 
on the second mode process. 

35. (Previously Presented) The program storage device of claim 29, wherein the 
instructions for performing the step of triggering comprise instructions for performing the steps 
of: 

executing first thread associated with the received command; and 
triggering a corresponding second thread to initiate the corresponding action by the 
second mode process. 
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36. (Original) The program storage device of claim 35, wherein the threads are applets. 

37. (Original) The program storage device of claim 35, w^herein the threads communicate 
via socket connections. 
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